{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SageMaker Benchmark Evaluation - Basic Usage\n",
    "\n",
    "This notebook demonstrates the basic user-facing flow for creating and managing benchmark evaluation jobs using the BenchmarkEvaluator with Jinja2 template-based pipeline generation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Discover Available Benchmarks\n",
    "\n",
    "Discover the benchmark properties and available options:\n",
    "https://docs.aws.amazon.com/sagemaker/latest/dg/nova-model-evaluation.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">&lt;</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">_Benchmark.MMLU:</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008700; text-decoration-color: #008700\">'mmlu'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000\">&lt;_Benchmark.MMLU_PRO: </span><span style=\"color: #008700; text-decoration-color: #008700\">'mmlu_pro'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000\">&lt;_Benchmark.BBH: </span><span style=\"color: #008700; text-decoration-color: #008700\">'bbh'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000\">&lt;_Benchmark.GPQA: </span><span style=\"color: #008700; text-decoration-color: #008700\">'gpqa'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000\">&lt;_Benchmark.MATH: </span><span style=\"color: #008700; text-decoration-color: #008700\">'math'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000\">&lt;_Benchmark.STRONG_REJECT: </span><span style=\"color: #008700; text-decoration-color: #008700\">'strong_reject'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000\">&lt;_Benchmark.IFEVAL: </span><span style=\"color: #008700; text-decoration-color: #008700\">'ifeval'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000\">&lt;_Benchmark.GEN_QA: </span><span style=\"color: #008700; text-decoration-color: #008700\">'gen_qa'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000\">&lt;_Benchmark.MMMU: </span><span style=\"color: #008700; text-decoration-color: #008700\">'mmmu'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000\">&lt;_Benchmark.LLM_JUDGE: </span><span style=\"color: #008700; text-decoration-color: #008700\">'llm_judge'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000\">&lt;_Benchmark.INFERENCE_ONLY: </span><span style=\"color: #008700; text-decoration-color: #008700\">'inference_only'</span><span style=\"font-weight: bold\">&gt;</span>\n",
       "<span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m<\u001b[0m\u001b[1;38;2;225;0;225m_Benchmark.MMLU:\u001b[0m\u001b[39m \u001b[0m\u001b[38;2;0;135;0m'mmlu'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[39m<_Benchmark.MMLU_PRO: \u001b[0m\u001b[38;2;0;135;0m'mmlu_pro'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[39m<_Benchmark.BBH: \u001b[0m\u001b[38;2;0;135;0m'bbh'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[39m<_Benchmark.GPQA: \u001b[0m\u001b[38;2;0;135;0m'gpqa'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[39m<_Benchmark.MATH: \u001b[0m\u001b[38;2;0;135;0m'math'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[39m<_Benchmark.STRONG_REJECT: \u001b[0m\u001b[38;2;0;135;0m'strong_reject'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[39m<_Benchmark.IFEVAL: \u001b[0m\u001b[38;2;0;135;0m'ifeval'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[39m<_Benchmark.GEN_QA: \u001b[0m\u001b[38;2;0;135;0m'gen_qa'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[39m<_Benchmark.MMMU: \u001b[0m\u001b[38;2;0;135;0m'mmmu'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[39m<_Benchmark.LLM_JUDGE: \u001b[0m\u001b[38;2;0;135;0m'llm_judge'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[39m<_Benchmark.INFERENCE_ONLY: \u001b[0m\u001b[38;2;0;135;0m'inference_only'\u001b[0m\u001b[1m>\u001b[0m\n",
       "\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'modality'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'Multi-Modal </span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">(</span><span style=\"color: #008700; text-decoration-color: #008700\">image</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">)</span><span style=\"color: #008700; text-decoration-color: #008700\">'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'description'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'Custom Dataset Evaluation – Lets you supply your own dataset for benchmarking, comparing model outputs to reference answers with metrics such as ROUGE and BLEU. gen_qa supports image inference for models which have multimodal support.'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'metrics'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008700; text-decoration-color: #008700\">'all'</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'strategy'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'gen_qa'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'subtask_available'</span>: <span style=\"color: #d70000; text-decoration-color: #d70000; font-style: italic\">False</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'subtasks'</span>: <span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'modality'\u001b[0m: \u001b[38;2;0;135;0m'Multi-Modal \u001b[0m\u001b[1;38;2;0;135;0m(\u001b[0m\u001b[38;2;0;135;0mimage\u001b[0m\u001b[1;38;2;0;135;0m)\u001b[0m\u001b[38;2;0;135;0m'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'description'\u001b[0m: \u001b[38;2;0;135;0m'Custom Dataset Evaluation – Lets you supply your own dataset for benchmarking, comparing model outputs to reference answers with metrics such as ROUGE and BLEU. gen_qa supports image inference for models which have multimodal support.'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'metrics'\u001b[0m: \u001b[1m[\u001b[0m\u001b[38;2;0;135;0m'all'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'strategy'\u001b[0m: \u001b[38;2;0;135;0m'gen_qa'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'subtask_available'\u001b[0m: \u001b[3;38;2;215;0;0mFalse\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'subtasks'\u001b[0m: \u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sagemaker.train.evaluate import get_benchmarks, get_benchmark_properties\n",
    "from rich.pretty import pprint\n",
    "\n",
    "# Configure logging to show INFO messages\n",
    "import logging\n",
    "logging.basicConfig(\n",
    "    level=logging.INFO,\n",
    "    format='%(levelname)s - %(name)s - %(message)s'\n",
    ")\n",
    "\n",
    "# Get available benchmarks\n",
    "Benchmark = get_benchmarks()\n",
    "pprint(list(Benchmark))\n",
    "\n",
    "# Print properties for a specific benchmark\n",
    "pprint(get_benchmark_properties(benchmark=Benchmark.GEN_QA))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Create BenchmarkEvaluator\n",
    "\n",
    "Create a BenchmarkEvaluator instance with the desired benchmark. The evaluator will use Jinja2 templates to render a complete pipeline definition.\n",
    "\n",
    "**Required Parameters:**\n",
    "- `benchmark`: Benchmark type from the Benchmark enum\n",
    "- `base_model`: Model ARN from SageMaker hub content\n",
    "- `output_s3_location`: S3 location for evaluation outputs\n",
    "- `mlflow_resource_arn`: MLflow tracking server ARN for experiment tracking\n",
    "\n",
    "**Optional Template Fields:**\n",
    "These fields are used for template rendering. If not provided, defaults will be used:\n",
    "- `model_package_group`: Model package group ARN\n",
    "- `source_model_package`: Source model package ARN\n",
    "- `dataset`: S3 URI of evaluation dataset\n",
    "- `model_artifact`: ARN of model artifact for lineage tracking (auto-inferred from source_model_package)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:39:45] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found credentials in shared credentials file: ~<span style=\"color: #e100e1; text-decoration-color: #e100e1\">/.aws/credentials</span>   <a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">credentials.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py#1364\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1364</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:39:45]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found credentials in shared credentials file: ~\u001b[38;2;225;0;225m/.aws/\u001b[0m\u001b[38;2;225;0;225mcredentials\u001b[0m   \u001b]8;id=314173;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py\u001b\\\u001b[2mcredentials.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=126855;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py#1364\u001b\\\u001b[2m1364\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /Library/Application Support/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /Users/mufi/Library/Application Support/sagemaker/config.yaml\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Resolved MLflow resource ARN:                                    <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#113\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">113</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         mmlu-eval-experiment                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Resolved MLflow resource ARN:                                    \u001b]8;id=480390;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=329695;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#113\u001b\\\u001b[2m113\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/ \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         mmlu-eval-experiment                                             \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Model package group provided as ARN:                             <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#145\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">145</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/exa <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         mple-name-aovqo                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Model package group provided as ARN:                             \u001b]8;id=572070;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=299487;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#145\u001b\\\u001b[2m145\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/exa \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         mple-name-aovqo                                                  \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">BenchMarkEvaluator</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">region</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">sagemaker_session</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">sagemaker.core.helper.session_helper.Session</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x13cd28e60</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">model</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">base_eval_name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'gen-qa-eval-demo'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">s3_output_path</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_resource_arn</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_experiment_name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_run_name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">networking</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">kms_key_id</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">model_package_group</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:model-package-group/example-name-aovqo'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">benchmark</span><span style=\"color: #000000; text-decoration-color: #000000\">=&lt;_Benchmark.GEN_QA: </span><span style=\"color: #008700; text-decoration-color: #008700\">'gen_qa'</span><span style=\"font-weight: bold\">&gt;</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">subtasks</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">dataset</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">evaluate_base_model</span>=<span style=\"color: #008700; text-decoration-color: #008700; font-style: italic\">True</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mBenchMarkEvaluator\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mregion\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0msagemaker_session\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;38;2;225;0;225msagemaker.core.helper.session_helper.Session\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x13cd28e60\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmodel\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mbase_eval_name\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'gen-qa-eval-demo'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0ms3_output_path\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_resource_arn\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_experiment_name\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_run_name\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mnetworking\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mkms_key_id\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmodel_package_group\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:model-package-group/example-name-aovqo'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mbenchmark\u001b[0m\u001b[39m=<_Benchmark.GEN_QA: \u001b[0m\u001b[38;2;0;135;0m'gen_qa'\u001b[0m\u001b[1m>\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0msubtasks\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mdataset\u001b[0m=\u001b[38;2;0;135;0m's3://sagemaker-us-west-2-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mevaluate_base_model\u001b[0m=\u001b[3;38;2;0;135;0mTrue\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sagemaker.train.evaluate import BenchMarkEvaluator\n",
    "\n",
    "# Create evaluator with GEN_QA benchmark\n",
    "# These values match our successfully tested configuration\n",
    "evaluator = BenchMarkEvaluator(\n",
    "    benchmark=Benchmark.GEN_QA,\n",
    "    model=\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28\",\n",
    "    s3_output_path=\"s3://mufi-test-serverless-smtj/eval/\",\n",
    "    mlflow_resource_arn=\"arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment\",\n",
    "    dataset=\"s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\",\n",
    "    model_package_group=\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/example-name-aovqo\", # Optional inferred from model if model package\n",
    "    base_eval_name=\"gen-qa-eval-demo\",\n",
    "    # Note: sagemaker_session is optional and will be auto-created if not provided\n",
    "    # Note: region is optional and will be auto deduced using environment variables - SAGEMAKER_REGION, AWS_REGION\n",
    ")\n",
    "\n",
    "pprint(evaluator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #ff0000; text-decoration-color: #ff0000\">╭─────────────────────────────── </span><span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold\">Traceback </span><span style=\"color: #ff7f7f; text-decoration-color: #ff7f7f; font-weight: bold\">(most recent call last)</span><span style=\"color: #ff0000; text-decoration-color: #ff0000\"> ────────────────────────────────╮</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> in &lt;module&gt;:13                                                                                   <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>                                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">10 # Create evaluator with GEN_QA benchmark</span>                                                    <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">11 # These values match our successfully tested configuration</span>                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">12 </span>evaluator = BenchMarkEvaluator(                                                             <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>13 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>benchmark=<span style=\"font-weight: bold; text-decoration: underline\">Benchmark</span>.GEN_QA,                                                             <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">14 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>model=<span style=\"color: #808000; text-decoration-color: #808000\">\"meta-textgeneration-llama-3-2-1b-instruct\"</span>,                                      <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">15 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>s3_output_path=<span style=\"color: #808000; text-decoration-color: #808000\">\"s3://mufi-test-serverless-smtj/eval/\"</span>,                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">16 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>mlflow_resource_arn=<span style=\"color: #808000; text-decoration-color: #808000\">\"arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server</span>    <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">╰──────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold\">NameError: </span>name <span style=\"color: #008700; text-decoration-color: #008700\">'Benchmark'</span> is not defined\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[38;2;255;0;0m╭─\u001b[0m\u001b[38;2;255;0;0m──────────────────────────────\u001b[0m\u001b[38;2;255;0;0m \u001b[0m\u001b[1;38;2;255;0;0mTraceback \u001b[0m\u001b[1;2;38;2;255;0;0m(most recent call last)\u001b[0m\u001b[38;2;255;0;0m \u001b[0m\u001b[38;2;255;0;0m───────────────────────────────\u001b[0m\u001b[38;2;255;0;0m─╮\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m in <module>:13                                                                                   \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m                                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m10 \u001b[0m\u001b[2m# Create evaluator with GEN_QA benchmark\u001b[0m                                                    \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m11 \u001b[0m\u001b[2m# These values match our successfully tested configuration\u001b[0m                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m12 \u001b[0mevaluator = BenchMarkEvaluator(                                                             \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[31m❱ \u001b[0m13 \u001b[2m│   \u001b[0mbenchmark=\u001b[1;4mBenchmark\u001b[0m.GEN_QA,                                                             \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m14 \u001b[0m\u001b[2m│   \u001b[0mmodel=\u001b[33m\"\u001b[0m\u001b[33mmeta-textgeneration-llama-3-2-1b-instruct\u001b[0m\u001b[33m\"\u001b[0m,                                      \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m15 \u001b[0m\u001b[2m│   \u001b[0ms3_output_path=\u001b[33m\"\u001b[0m\u001b[33ms3://mufi-test-serverless-smtj/eval/\u001b[0m\u001b[33m\"\u001b[0m,                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m16 \u001b[0m\u001b[2m│   \u001b[0mmlflow_resource_arn=\u001b[33m\"\u001b[0m\u001b[33marn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server\u001b[0m    \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
       "\u001b[1;91mNameError: \u001b[0mname \u001b[38;2;0;135;0m'Benchmark'\u001b[0m is not defined\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# # [Optional] BASE MODEL EVAL\n",
    "\n",
    "# from sagemaker.train.evaluate import BenchMarkEvaluator\n",
    "\n",
    "# # Create evaluator with GEN_QA benchmark\n",
    "# # These values match our successfully tested configuration\n",
    "# evaluator = BenchMarkEvaluator(\n",
    "#     benchmark=Benchmark.GEN_QA,\n",
    "#     model=\"meta-textgeneration-llama-3-2-1b-instruct\",\n",
    "#     s3_output_path=\"s3://mufi-test-serverless-smtj/eval/\",\n",
    "#     mlflow_resource_arn=\"arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment\",\n",
    "#     dataset=\"s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\",\n",
    "#     # model_package_group=\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/example-name-aovqo\", # Optional inferred from model if model package\n",
    "#     base_eval_name=\"gen-qa-eval-demo\",\n",
    "#     # Note: sagemaker_session is optional and will be auto-created if not provided\n",
    "#     # Note: region is optional and will be auto deduced using environment variables - SAGEMAKER_REGION, AWS_REGION\n",
    "# )\n",
    "\n",
    "# pprint(evaluator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials\n",
      "INFO - sagemaker.modules.evaluate.base_evaluator - Model package group provided as ARN: arn:aws:sagemaker:us-east-1:052150106756:model-package-group/test-nova-finetuned-models\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">BenchMarkEvaluator</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">region</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'us-east-1'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">sagemaker_session</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">sagemaker_core.helper.session_helper.Session</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x356a03950</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">model</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-east-1:052150106756:model-package/test-nova-finetuned-models/3'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">base_eval_name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'gen-qa-eval-demo'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">s3_output_path</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-iad/eval/'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_resource_arn</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-east-1:052150106756:mlflow-tracking-server/mlflow-prod-server'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_experiment_name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_run_name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">networking</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">kms_key_id</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">model_package_group</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-east-1:052150106756:model-package-group/test-nova-finetuned-models'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">benchmark</span><span style=\"color: #000000; text-decoration-color: #000000\">=&lt;_Benchmark.GEN_QA: </span><span style=\"color: #008700; text-decoration-color: #008700\">'gen_qa'</span><span style=\"font-weight: bold\">&gt;</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">subtasks</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">dataset</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'s3://sagemaker-us-east-1-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">evaluate_base_model</span>=<span style=\"color: #008700; text-decoration-color: #008700; font-style: italic\">True</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mBenchMarkEvaluator\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mregion\u001b[0m=\u001b[38;2;0;135;0m'us-east-1'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0msagemaker_session\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;38;2;225;0;225msagemaker_core.helper.session_helper.Session\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x356a03950\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmodel\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-east-1:052150106756:model-package/test-nova-finetuned-models/3'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mbase_eval_name\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'gen-qa-eval-demo'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0ms3_output_path\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m's3://mufi-test-serverless-iad/eval/'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_resource_arn\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-east-1:052150106756:mlflow-tracking-server/mlflow-prod-server'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_experiment_name\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_run_name\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mnetworking\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mkms_key_id\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmodel_package_group\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-east-1:052150106756:model-package-group/test-nova-finetuned-models'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mbenchmark\u001b[0m\u001b[39m=<_Benchmark.GEN_QA: \u001b[0m\u001b[38;2;0;135;0m'gen_qa'\u001b[0m\u001b[1m>\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0msubtasks\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mdataset\u001b[0m=\u001b[38;2;0;135;0m's3://sagemaker-us-east-1-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mevaluate_base_model\u001b[0m=\u001b[3;38;2;0;135;0mTrue\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# # [Optional] Nova testing IAD Prod\n",
    "\n",
    "# from sagemaker.train.evaluate import BenchMarkEvaluator\n",
    "\n",
    "# # Create evaluator with GEN_QA benchmark\n",
    "# # These values match our successfully tested configuration\n",
    "# evaluator = BenchMarkEvaluator(\n",
    "#     benchmark=Benchmark.GEN_QA,\n",
    "#     # model=\"arn:aws:sagemaker:us-east-1:052150106756:model-package/bgrv-nova-micro-sft-lora/1\",\n",
    "#     model=\"arn:aws:sagemaker:us-east-1:052150106756:model-package/test-nova-finetuned-models/3\",\n",
    "#     s3_output_path=\"s3://mufi-test-serverless-iad/eval/\",\n",
    "#     mlflow_resource_arn=\"arn:aws:sagemaker:us-east-1:052150106756:mlflow-tracking-server/mlflow-prod-server\",\n",
    "#     dataset=\"s3://sagemaker-us-east-1-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\",\n",
    "#     model_package_group=\"arn:aws:sagemaker:us-east-1:052150106756:model-package-group/test-nova-finetuned-models\", # Optional inferred from model if model package\n",
    "#     base_eval_name=\"gen-qa-eval-demo\",\n",
    "#     region=\"us-east-1\",\n",
    "#     # Note: sagemaker_session is optional and will be auto-created if not provided\n",
    "#     # Note: region is optional and will be auto deduced using environment variables - SAGEMAKER_REGION, AWS_REGION\n",
    "# )\n",
    "\n",
    "# pprint(evaluator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optionally update the hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:26:31] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> SageMaker Python SDK will collect telemetry to help us better  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">telemetry_logging.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">91</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         understand our user's needs, diagnose issues, and deliver      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         additional features.                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         To opt out of telemetry, please disable via TelemetryOptOut    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         parameter in SDK defaults config. For more information, refer  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         to                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">https://sagemaker.readthedocs.io/en/stable/overview.html#confi</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">guring-and-using-defaults-with-the-sagemaker-python-sdk.</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:26:31]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m SageMaker Python SDK will collect telemetry to help us better  \u001b]8;id=665742;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=28065;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         understand our user's needs, diagnose issues, and deliver      \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         additional features.                                           \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         To opt out of telemetry, please disable via TelemetryOptOut    \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         parameter in SDK defaults config. For more information, refer  \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         to                                                             \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#confi\u001b[0m \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m       \u001b[2m                       \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Fetching evaluation override parameters for hyperparameters <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">benchmark_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py#495\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">495</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         property                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Fetching evaluation override parameters for hyperparameters \u001b]8;id=668827;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py\u001b\\\u001b[2mbenchmark_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=344195;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py#495\u001b\\\u001b[2m495\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         property                                                    \u001b[2m                          \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Fetching hub content metadata for                                  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">recipe_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#201\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">201</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         meta-textgeneration-llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>-1b-instruct from SageMakerPublicHub  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Fetching hub content metadata for                                  \u001b]8;id=912465;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\u001b\\\u001b[2mrecipe_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=530916;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#201\u001b\\\u001b[2m201\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         meta-textgeneration-llama-\u001b[1;36m3\u001b[0m-\u001b[1;36m2\u001b[0m-1b-instruct from SageMakerPublicHub  \u001b[2m                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #d7af00; text-decoration-color: #d7af00; font-weight: bold\">WARNING </span> No region provided. Using default region.                                 <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py#340\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">340</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m No region provided. Using default region.                                 \u001b]8;id=483608;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=394176;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py#340\u001b\\\u001b[2m340\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Runs on sagemaker us-west-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, region:us-west-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>                             <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py#354\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">354</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Runs on sagemaker us-west-\u001b[1;36m2\u001b[0m, region:us-west-\u001b[1;36m2\u001b[0m                             \u001b]8;id=127187;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=740445;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py#354\u001b\\\u001b[2m354\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for evaluation recipe with <span style=\"color: #d7af00; text-decoration-color: #d7af00\">Type</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Evaluation'</span> and         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">recipe_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#221\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">221</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #d7af00; text-decoration-color: #d7af00\">EvaluationType</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'DeterministicEvaluation'</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for evaluation recipe with \u001b[38;2;215;175;0mType\u001b[0m=\u001b[38;2;0;135;0m'Evaluation'\u001b[0m and         \u001b]8;id=26417;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\u001b\\\u001b[2mrecipe_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=309515;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#221\u001b\\\u001b[2m221\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;215;175;0mEvaluationType\u001b[0m=\u001b[38;2;0;135;0m'DeterministicEvaluation'\u001b[0m                           \u001b[2m                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Downloading override parameters from                               <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">recipe_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#249\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">249</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//jumpstart-cache-beta-us-west-2/recipes/open-source-eval-meta-</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">textgeneration-llama-3-2-1b-instruct-deterministic_override_params</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">_sm_jobs_v1.0.19.json</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Downloading override parameters from                               \u001b]8;id=762738;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\u001b\\\u001b[2mrecipe_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=1149;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#249\u001b\\\u001b[2m249\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/jumpstart-cache-beta-us-west-2/recipes/\u001b[0m\u001b[38;2;225;0;225mopen-source-eval-meta-\u001b[0m \u001b[2m                   \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225mtextgeneration-llama-3-2-1b-instruct-deterministic_override_params\u001b[0m \u001b[2m                   \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225m_sm_jobs_v1.0.19.json\u001b[0m                                              \u001b[2m                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'max_new_tokens'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'8192'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'temperature'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'0'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'top_k'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'-1'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'top_p'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'1.0'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'aggregation'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">''</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'postprocessing'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'False'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'max_model_len'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'12000'</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'max_new_tokens'\u001b[0m: \u001b[38;2;0;135;0m'8192'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'temperature'\u001b[0m: \u001b[38;2;0;135;0m'0'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'top_k'\u001b[0m: \u001b[38;2;0;135;0m'-1'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'top_p'\u001b[0m: \u001b[38;2;0;135;0m'1.0'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'aggregation'\u001b[0m: \u001b[38;2;0;135;0m''\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'postprocessing'\u001b[0m: \u001b[38;2;0;135;0m'False'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'max_model_len'\u001b[0m: \u001b[38;2;0;135;0m'12000'\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pprint(evaluator.hyperparameters.to_dict())\n",
    "\n",
    "# optionally update hyperparameters\n",
    "# evaluator.hyperparameters.temperature = \"0.1\"\n",
    "\n",
    "# optionally get more info on types, limits, defaults.\n",
    "# evaluator.hyperparameters.get_info()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Run Evaluation\n",
    "\n",
    "Start a benchmark evaluation job. The system will:\n",
    "1. Build template context with all required parameters\n",
    "2. Render the pipeline definition from `DETERMINISTIC_TEMPLATE` using Jinja2\n",
    "3. Create or update the pipeline with the rendered definition\n",
    "4. Start the pipeline execution with empty parameters (all values pre-substituted)\n",
    "\n",
    "**What happens during execution:**\n",
    "- CreateEvaluationAction: Sets up lineage tracking\n",
    "- EvaluateBaseModel & EvaluateCustomModel: Run in parallel as serverless training jobs\n",
    "- AssociateLineage: Links evaluation results to lineage tracking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:40:20] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> SageMaker Python SDK will collect telemetry to help us better  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">telemetry_logging.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">91</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         understand our user's needs, diagnose issues, and deliver      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         additional features.                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         To opt out of telemetry, please disable via TelemetryOptOut    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         parameter in SDK defaults config. For more information, refer  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         to                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">https://sagemaker.readthedocs.io/en/stable/overview.html#confi</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">guring-and-using-defaults-with-the-sagemaker-python-sdk.</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:40:20]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m SageMaker Python SDK will collect telemetry to help us better  \u001b]8;id=39435;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=899931;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         understand our user's needs, diagnose issues, and deliver      \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         additional features.                                           \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         To opt out of telemetry, please disable via TelemetryOptOut    \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         parameter in SDK defaults config. For more information, refer  \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         to                                                             \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#confi\u001b[0m \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m       \u001b[2m                       \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Getting or creating artifact for source:                         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#597\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">597</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         tuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Getting or creating artifact for source:                         \u001b]8;id=774478;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=222956;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#597\u001b\\\u001b[2m597\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         tuned-models-gamma/\u001b[1;36m28\u001b[0m                                            \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for existing artifact for model package:               <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#459\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">459</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         tuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for existing artifact for model package:               \u001b]8;id=672788;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=533927;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#459\u001b\\\u001b[2m459\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         tuned-models-gamma/\u001b[1;36m28\u001b[0m                                            \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found existing artifact:                                         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#468\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">468</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b3 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         138877d772ec489bef                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found existing artifact:                                         \u001b]8;id=555230;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=311641;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#468\u001b\\\u001b[2m468\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b3 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         138877d772ec489bef                                               \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using resolved model_package_group ARN:                          <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#414\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">414</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/exa <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         mple-name-aovqo                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using resolved model_package_group ARN:                          \u001b]8;id=350625;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=393598;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#414\u001b\\\u001b[2m414\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/exa \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         mple-name-aovqo                                                  \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using ModelPackage - model_package_group_arn:               <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">benchmark_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py#644\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">644</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package-grou <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         p/example-name-aovqo                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using ModelPackage - model_package_group_arn:               \u001b]8;id=534430;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py\u001b\\\u001b[2mbenchmark_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=895229;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py#644\u001b\\\u001b[2m644\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package-grou \u001b[2m                          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         p/example-name-aovqo                                        \u001b[2m                          \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Resolved model info - base_model_name:                      <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">benchmark_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py#647\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">647</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         meta-textgeneration-llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>-1b-instruct, base_model_arn:  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublic <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         Hub/Model/meta-textgeneration-llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>-1b-instruct/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.10</span>.<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>          source_model_package_arn:                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/test <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         -finetuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Resolved model info - base_model_name:                      \u001b]8;id=1084;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py\u001b\\\u001b[2mbenchmark_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=849460;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py#647\u001b\\\u001b[2m647\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         meta-textgeneration-llama-\u001b[1;36m3\u001b[0m-\u001b[1;36m2\u001b[0m-1b-instruct, base_model_arn:  \u001b[2m                          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublic \u001b[2m                          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         Hub/Model/meta-textgeneration-llama-\u001b[1;36m3\u001b[0m-\u001b[1;36m2\u001b[0m-1b-instruct/\u001b[1;36m1.10\u001b[0m.\u001b[1;36m0\u001b[0m, \u001b[2m                          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m          source_model_package_arn:                                  \u001b[2m                          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/test \u001b[2m                          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         -finetuned-models-gamma/\u001b[1;36m28\u001b[0m                                  \u001b[2m                          \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> SageMaker Python SDK will collect telemetry to help us better  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">telemetry_logging.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">91</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         understand our user's needs, diagnose issues, and deliver      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         additional features.                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         To opt out of telemetry, please disable via TelemetryOptOut    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         parameter in SDK defaults config. For more information, refer  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         to                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">https://sagemaker.readthedocs.io/en/stable/overview.html#confi</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">guring-and-using-defaults-with-the-sagemaker-python-sdk.</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m SageMaker Python SDK will collect telemetry to help us better  \u001b]8;id=537782;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=387290;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         understand our user's needs, diagnose issues, and deliver      \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         additional features.                                           \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         To opt out of telemetry, please disable via TelemetryOptOut    \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         parameter in SDK defaults config. For more information, refer  \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         to                                                             \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#confi\u001b[0m \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m       \u001b[2m                       \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Fetching evaluation override parameters for hyperparameters <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">benchmark_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py#495\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">495</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         property                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Fetching evaluation override parameters for hyperparameters \u001b]8;id=706064;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py\u001b\\\u001b[2mbenchmark_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=284205;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py#495\u001b\\\u001b[2m495\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         property                                                    \u001b[2m                          \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Fetching hub content metadata for                                  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">recipe_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#201\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">201</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         meta-textgeneration-llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>-1b-instruct from SageMakerPublicHub  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Fetching hub content metadata for                                  \u001b]8;id=502448;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\u001b\\\u001b[2mrecipe_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=531984;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#201\u001b\\\u001b[2m201\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         meta-textgeneration-llama-\u001b[1;36m3\u001b[0m-\u001b[1;36m2\u001b[0m-1b-instruct from SageMakerPublicHub  \u001b[2m                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for evaluation recipe with <span style=\"color: #d7af00; text-decoration-color: #d7af00\">Type</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Evaluation'</span> and         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">recipe_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#221\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">221</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #d7af00; text-decoration-color: #d7af00\">EvaluationType</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'DeterministicEvaluation'</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for evaluation recipe with \u001b[38;2;215;175;0mType\u001b[0m=\u001b[38;2;0;135;0m'Evaluation'\u001b[0m and         \u001b]8;id=67072;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\u001b\\\u001b[2mrecipe_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=119115;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#221\u001b\\\u001b[2m221\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;215;175;0mEvaluationType\u001b[0m=\u001b[38;2;0;135;0m'DeterministicEvaluation'\u001b[0m                           \u001b[2m                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Downloading override parameters from                               <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">recipe_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#249\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">249</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//jumpstart-cache-beta-us-west-2/recipes/open-source-eval-meta-</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">textgeneration-llama-3-2-1b-instruct-deterministic_override_params</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">_sm_jobs_v1.0.19.json</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Downloading override parameters from                               \u001b]8;id=954396;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\u001b\\\u001b[2mrecipe_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=959350;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#249\u001b\\\u001b[2m249\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/jumpstart-cache-beta-us-west-2/recipes/\u001b[0m\u001b[38;2;225;0;225mopen-source-eval-meta-\u001b[0m \u001b[2m                   \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225mtextgeneration-llama-3-2-1b-instruct-deterministic_override_params\u001b[0m \u001b[2m                   \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225m_sm_jobs_v1.0.19.json\u001b[0m                                              \u001b[2m                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:40:21] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using configured hyperparameters: <span style=\"font-weight: bold\">{</span><span style=\"color: #008700; text-decoration-color: #008700\">'max_new_tokens'</span>:        <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">benchmark_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py#568\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">568</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'8192'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'temperature'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'0'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'top_k'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'-1'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'top_p'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'1.0'</span>,  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'aggregation'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">''</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'postprocessing'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'False'</span>,               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'max_model_len'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'12000'</span><span style=\"font-weight: bold\">}</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                          </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:40:21]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using configured hyperparameters: \u001b[1m{\u001b[0m\u001b[38;2;0;135;0m'max_new_tokens'\u001b[0m:        \u001b]8;id=584498;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py\u001b\\\u001b[2mbenchmark_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=126531;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/benchmark_evaluator.py#568\u001b\\\u001b[2m568\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'8192'\u001b[0m, \u001b[38;2;0;135;0m'temperature'\u001b[0m: \u001b[38;2;0;135;0m'0'\u001b[0m, \u001b[38;2;0;135;0m'top_k'\u001b[0m: \u001b[38;2;0;135;0m'-1'\u001b[0m, \u001b[38;2;0;135;0m'top_p'\u001b[0m: \u001b[38;2;0;135;0m'1.0'\u001b[0m,  \u001b[2m                          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'aggregation'\u001b[0m: \u001b[38;2;0;135;0m''\u001b[0m, \u001b[38;2;0;135;0m'postprocessing'\u001b[0m: \u001b[38;2;0;135;0m'False'\u001b[0m,               \u001b[2m                          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'max_model_len'\u001b[0m: \u001b[38;2;0;135;0m'12000'\u001b[0m\u001b[1m}\u001b[0m                                   \u001b[2m                          \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using full template for ModelPackage                             <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#655\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">655</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using full template for ModelPackage                             \u001b]8;id=556396;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=773270;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#655\u001b\\\u001b[2m655\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Resolved template parameters: <span style=\"font-weight: bold\">{</span><span style=\"color: #008700; text-decoration-color: #008700\">'role_arn'</span>:                       <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#693\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">693</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:iam::052150106756:role/Admin'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'mlflow_resource_arn'</span>:   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">/mmlu-eval-experiment'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'mlflow_experiment_name'</span>: <span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'mlflow_run_name'</span>: <span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'model_package_group_arn'</span>:              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:model-package-group/ex</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">ample-name-aovqo'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'source_model_package_arn'</span>:                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'base_model_arn'</span>:                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">Model/meta-textgeneration-llama-3-2-1b-instruct/1.10.0'</span>,         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'s3_output_path'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/'</span>,        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'dataset_artifact_arn'</span>:                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">3138877d772ec489bef'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'action_arn_prefix'</span>:                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:action'</span>,               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'dataset_uri'</span>:                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t19</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">5443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'task'</span>:   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'gen_qa'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'strategy'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'gen_qa'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'evaluation_metric'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'all'</span>,      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'subtask'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">''</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'pipeline_name'</span>:                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'SagemakerEvaluation-Deterministic'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'evaluate_base_model'</span>:      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700; font-style: italic\">True</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'max_new_tokens'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'8192'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'temperature'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'0'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'top_k'</span>:     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'-1'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'top_p'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'1.0'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'aggregation'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">''</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'postprocessing'</span>:       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'False'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'max_model_len'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'12000'</span><span style=\"font-weight: bold\">}</span>                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Resolved template parameters: \u001b[1m{\u001b[0m\u001b[38;2;0;135;0m'role_arn'\u001b[0m:                       \u001b]8;id=970601;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=386360;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#693\u001b\\\u001b[2m693\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:iam::052150106756:role/Admin'\u001b[0m, \u001b[38;2;0;135;0m'mlflow_resource_arn'\u001b[0m:   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m/mmlu-eval-experiment'\u001b[0m, \u001b[38;2;0;135;0m'mlflow_experiment_name'\u001b[0m: \u001b[3;38;2;225;0;225mNone\u001b[0m,          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'mlflow_run_name'\u001b[0m: \u001b[3;38;2;225;0;225mNone\u001b[0m, \u001b[38;2;0;135;0m'model_package_group_arn'\u001b[0m:              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:model-package-group/ex\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mample-name-aovqo'\u001b[0m, \u001b[38;2;0;135;0m'source_model_package_arn'\u001b[0m:                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28'\u001b[0m, \u001b[38;2;0;135;0m'base_model_arn'\u001b[0m:                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mModel/meta-textgeneration-llama-3-2-1b-instruct/1.10.0'\u001b[0m,         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m's3_output_path'\u001b[0m: \u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/'\u001b[0m,        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'dataset_artifact_arn'\u001b[0m:                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m3138877d772ec489bef'\u001b[0m, \u001b[38;2;0;135;0m'action_arn_prefix'\u001b[0m:                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:action'\u001b[0m,               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'dataset_uri'\u001b[0m:                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m's3://sagemaker-us-west-2-052150106756/studio-users/d20251107t19\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m5443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl'\u001b[0m, \u001b[38;2;0;135;0m'task'\u001b[0m:   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'gen_qa'\u001b[0m, \u001b[38;2;0;135;0m'strategy'\u001b[0m: \u001b[38;2;0;135;0m'gen_qa'\u001b[0m, \u001b[38;2;0;135;0m'evaluation_metric'\u001b[0m: \u001b[38;2;0;135;0m'all'\u001b[0m,      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'subtask'\u001b[0m: \u001b[38;2;0;135;0m''\u001b[0m, \u001b[38;2;0;135;0m'pipeline_name'\u001b[0m:                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'SagemakerEvaluation-Deterministic'\u001b[0m, \u001b[38;2;0;135;0m'evaluate_base_model'\u001b[0m:      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[3;38;2;0;135;0mTrue\u001b[0m, \u001b[38;2;0;135;0m'max_new_tokens'\u001b[0m: \u001b[38;2;0;135;0m'8192'\u001b[0m, \u001b[38;2;0;135;0m'temperature'\u001b[0m: \u001b[38;2;0;135;0m'0'\u001b[0m, \u001b[38;2;0;135;0m'top_k'\u001b[0m:     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'-1'\u001b[0m, \u001b[38;2;0;135;0m'top_p'\u001b[0m: \u001b[38;2;0;135;0m'1.0'\u001b[0m, \u001b[38;2;0;135;0m'aggregation'\u001b[0m: \u001b[38;2;0;135;0m''\u001b[0m, \u001b[38;2;0;135;0m'postprocessing'\u001b[0m:       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'False'\u001b[0m, \u001b[38;2;0;135;0m'max_model_len'\u001b[0m: \u001b[38;2;0;135;0m'12000'\u001b[0m\u001b[1m}\u001b[0m                               \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Rendered pipeline definition:                                    <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#702\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">702</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"font-weight: bold\">{</span>                                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Version\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"2020-12-01\"</span>,                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Metadata\"</span>: <span style=\"font-weight: bold\">{}</span>,                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"MlflowConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"color: #008700; text-decoration-color: #008700\">\"MlflowResourceArn\"</span>:                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">/mmlu-eval-experiment\"</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"font-weight: bold\">}</span>,                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Parameters\"</span>: <span style=\"font-weight: bold\">[]</span>,                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Steps\"</span>: <span style=\"font-weight: bold\">[</span>                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"CreateEvaluationAction\"</span>,                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Lineage\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Actions\"</span>: <span style=\"font-weight: bold\">[</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ActionName\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ActionType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Evaluation\"</span>,                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>:                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackage\"</span>                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Properties\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"PipelineExecutionArn\"</span>: <span style=\"font-weight: bold\">{</span>                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionArn\"</span>          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"PipelineName\"</span>:                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"SagemakerEvaluation-Deterministic\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Contexts\"</span>: <span style=\"font-weight: bold\">[</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ContextName\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ContextType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"PipelineExecution\"</span>,                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionArn\"</span>          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Associations\"</span>: <span style=\"font-weight: bold\">[</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Action\"</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Context\"</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>,                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>:                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">3138877d772ec489bef\"</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>: <span style=\"font-weight: bold\">{</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"/\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:ac</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">tion\"</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluateBaseModel\"</span>,                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Training\"</span>,                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"RoleArn\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:iam::052150106756:role/Admin\"</span>,       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackageConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackageGroupArn\"</span>:                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/ex</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">ample-name-aovqo\"</span>,                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceModelPackageArn\"</span>:                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28\"</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"ServerlessJobConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"BaseModelArn\"</span>:                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">Model/meta-textgeneration-llama-3-2-1b-instruct/1.10.0\"</span>,         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"AcceptEula\"</span>: true,                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"JobType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Evaluation\"</span>,                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"BenchmarkEvaluation\"</span>                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"StoppingCondition\"</span>: <span style=\"font-weight: bold\">{</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"MaxRuntimeInSeconds\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">86400</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"HyperParameters\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"task\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"gen_qa\"</span>,                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"strategy\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"gen_qa\"</span>,                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"evaluation_metric\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"all\"</span>,                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"max_new_tokens\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"8192\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"temperature\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"0\"</span>,                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"top_k\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-1\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"top_p\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"1.0\"</span>,                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"max_model_len\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"12000\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"aggregation\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"postprocessing\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"False\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"OutputDataConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"S3OutputPath\"</span>:                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://mufi-test-serverless-smtj/eval/\"</span>,                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"CompressionType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"NONE\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"InputDataConfig\"</span>: <span style=\"font-weight: bold\">[</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ChannelName\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"train\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"DataSource\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"S3DataSource\"</span>: <span style=\"font-weight: bold\">{</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"S3DataType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"S3Prefix\"</span>,                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"S3Uri\"</span>:                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t19</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">5443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\"</span>            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluateCustomModel\"</span>,                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Training\"</span>,                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"RoleArn\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:iam::052150106756:role/Admin\"</span>,       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackageConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackageGroupArn\"</span>:                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/ex</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">ample-name-aovqo\"</span>,                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceModelPackageArn\"</span>:                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28\"</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"ServerlessJobConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"BaseModelArn\"</span>:                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">Model/meta-textgeneration-llama-3-2-1b-instruct/1.10.0\"</span>,         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"AcceptEula\"</span>: true,                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"JobType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Evaluation\"</span>,                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"BenchmarkEvaluation\"</span>                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"StoppingCondition\"</span>: <span style=\"font-weight: bold\">{</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"MaxRuntimeInSeconds\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">86400</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"HyperParameters\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"task\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"gen_qa\"</span>,                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"strategy\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"gen_qa\"</span>,                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"evaluation_metric\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"all\"</span>,                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"max_new_tokens\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"8192\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"temperature\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"0\"</span>,                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"top_k\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-1\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"top_p\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"1.0\"</span>,                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"max_model_len\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"12000\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"aggregation\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"postprocessing\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"False\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"OutputDataConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"S3OutputPath\"</span>:                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://mufi-test-serverless-smtj/eval/\"</span>,                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"CompressionType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"NONE\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"InputDataConfig\"</span>: <span style=\"font-weight: bold\">[</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ChannelName\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"train\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"DataSource\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"S3DataSource\"</span>: <span style=\"font-weight: bold\">{</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"S3DataType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"S3Prefix\"</span>,                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"S3Uri\"</span>:                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t19</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">5443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\"</span>            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociateLineage\"</span>,                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Lineage\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"DependsOn\"</span>: <span style=\"font-weight: bold\">[</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"CreateEvaluationAction\"</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">]</span>,                                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Artifacts\"</span>: <span style=\"font-weight: bold\">[</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ArtifactName\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">{</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">}</span>,                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"base-eval-report\"</span>                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">]</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ArtifactType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluationReport\"</span>,                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>:                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"Steps.EvaluateBaseModel.OutputDataConfig.S3OutputPath\"</span>          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>,                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ArtifactName\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">{</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">}</span>,                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"custom-eval-report\"</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">]</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ArtifactType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluationReport\"</span>,                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>:                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"Steps.EvaluateCustomModel.OutputDataConfig.S3OutputPath\"</span>        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Associations\"</span>: <span style=\"font-weight: bold\">[</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>,                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"base-eval-report\"</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Artifact\"</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>: <span style=\"font-weight: bold\">{</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"/\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:ac</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">tion\"</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>,                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>,                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"custom-eval-report\"</span>                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Artifact\"</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>: <span style=\"font-weight: bold\">{</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"/\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:ac</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">tion\"</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"font-weight: bold\">]</span>                                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"font-weight: bold\">}</span>                                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Rendered pipeline definition:                                    \u001b]8;id=330131;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=262009;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#702\u001b\\\u001b[2m702\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[1m{\u001b[0m                                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Version\"\u001b[0m: \u001b[38;2;0;135;0m\"2020-12-01\"\u001b[0m,                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Metadata\"\u001b[0m: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m,                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"MlflowConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[38;2;0;135;0m\"MlflowResourceArn\"\u001b[0m:                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m/mmlu-eval-experiment\"\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[1m}\u001b[0m,                                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Parameters\"\u001b[0m: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m,                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Steps\"\u001b[0m: \u001b[1m[\u001b[0m                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"CreateEvaluationAction\"\u001b[0m,                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Lineage\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Actions\"\u001b[0m: \u001b[1m[\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ActionName\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ActionType\"\u001b[0m: \u001b[38;2;0;135;0m\"Evaluation\"\u001b[0m,                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m:                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceType\"\u001b[0m: \u001b[38;2;0;135;0m\"ModelPackage\"\u001b[0m                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Properties\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"PipelineExecutionArn\"\u001b[0m: \u001b[1m{\u001b[0m                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionArn\"\u001b[0m          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"PipelineName\"\u001b[0m:                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"SagemakerEvaluation-Deterministic\"\u001b[0m                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Contexts\"\u001b[0m: \u001b[1m[\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ContextName\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ContextType\"\u001b[0m: \u001b[38;2;0;135;0m\"PipelineExecution\"\u001b[0m,                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionArn\"\u001b[0m          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Associations\"\u001b[0m: \u001b[1m[\u001b[0m                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Action\"\u001b[0m                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Context\"\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m,                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m:                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m3138877d772ec489bef\"\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m: \u001b[1m{\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"/\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:ac\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mtion\"\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"EvaluateBaseModel\"\u001b[0m,                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Training\"\u001b[0m,                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"RoleArn\"\u001b[0m: \u001b[38;2;0;135;0m\"arn:aws:iam::052150106756:role/Admin\"\u001b[0m,       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"ModelPackageConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"ModelPackageGroupArn\"\u001b[0m:                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/ex\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mample-name-aovqo\"\u001b[0m,                                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"SourceModelPackageArn\"\u001b[0m:                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28\"\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"ServerlessJobConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"BaseModelArn\"\u001b[0m:                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mModel/meta-textgeneration-llama-3-2-1b-instruct/1.10.0\"\u001b[0m,         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"AcceptEula\"\u001b[0m: true,                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"JobType\"\u001b[0m: \u001b[38;2;0;135;0m\"Evaluation\"\u001b[0m,                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"EvaluationType\"\u001b[0m: \u001b[38;2;0;135;0m\"BenchmarkEvaluation\"\u001b[0m                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"StoppingCondition\"\u001b[0m: \u001b[1m{\u001b[0m                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"MaxRuntimeInSeconds\"\u001b[0m: \u001b[1;36m86400\u001b[0m                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"HyperParameters\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"task\"\u001b[0m: \u001b[38;2;0;135;0m\"gen_qa\"\u001b[0m,                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"strategy\"\u001b[0m: \u001b[38;2;0;135;0m\"gen_qa\"\u001b[0m,                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"evaluation_metric\"\u001b[0m: \u001b[38;2;0;135;0m\"all\"\u001b[0m,                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"max_new_tokens\"\u001b[0m: \u001b[38;2;0;135;0m\"8192\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"temperature\"\u001b[0m: \u001b[38;2;0;135;0m\"0\"\u001b[0m,                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"top_k\"\u001b[0m: \u001b[38;2;0;135;0m\"-1\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"top_p\"\u001b[0m: \u001b[38;2;0;135;0m\"1.0\"\u001b[0m,                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"max_model_len\"\u001b[0m: \u001b[38;2;0;135;0m\"12000\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"aggregation\"\u001b[0m: \u001b[38;2;0;135;0m\"\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"postprocessing\"\u001b[0m: \u001b[38;2;0;135;0m\"False\"\u001b[0m                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"OutputDataConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"S3OutputPath\"\u001b[0m:                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"s3://mufi-test-serverless-smtj/eval/\"\u001b[0m,                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"CompressionType\"\u001b[0m: \u001b[38;2;0;135;0m\"NONE\"\u001b[0m                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"InputDataConfig\"\u001b[0m: \u001b[1m[\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ChannelName\"\u001b[0m: \u001b[38;2;0;135;0m\"train\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"DataSource\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"S3DataSource\"\u001b[0m: \u001b[1m{\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"S3DataType\"\u001b[0m: \u001b[38;2;0;135;0m\"S3Prefix\"\u001b[0m,                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"S3Uri\"\u001b[0m:                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t19\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m5443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\"\u001b[0m            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"EvaluateCustomModel\"\u001b[0m,                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Training\"\u001b[0m,                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"RoleArn\"\u001b[0m: \u001b[38;2;0;135;0m\"arn:aws:iam::052150106756:role/Admin\"\u001b[0m,       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"ModelPackageConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"ModelPackageGroupArn\"\u001b[0m:                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/ex\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mample-name-aovqo\"\u001b[0m,                                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"SourceModelPackageArn\"\u001b[0m:                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28\"\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"ServerlessJobConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"BaseModelArn\"\u001b[0m:                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mModel/meta-textgeneration-llama-3-2-1b-instruct/1.10.0\"\u001b[0m,         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"AcceptEula\"\u001b[0m: true,                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"JobType\"\u001b[0m: \u001b[38;2;0;135;0m\"Evaluation\"\u001b[0m,                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"EvaluationType\"\u001b[0m: \u001b[38;2;0;135;0m\"BenchmarkEvaluation\"\u001b[0m                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"StoppingCondition\"\u001b[0m: \u001b[1m{\u001b[0m                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"MaxRuntimeInSeconds\"\u001b[0m: \u001b[1;36m86400\u001b[0m                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"HyperParameters\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"task\"\u001b[0m: \u001b[38;2;0;135;0m\"gen_qa\"\u001b[0m,                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"strategy\"\u001b[0m: \u001b[38;2;0;135;0m\"gen_qa\"\u001b[0m,                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"evaluation_metric\"\u001b[0m: \u001b[38;2;0;135;0m\"all\"\u001b[0m,                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"max_new_tokens\"\u001b[0m: \u001b[38;2;0;135;0m\"8192\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"temperature\"\u001b[0m: \u001b[38;2;0;135;0m\"0\"\u001b[0m,                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"top_k\"\u001b[0m: \u001b[38;2;0;135;0m\"-1\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"top_p\"\u001b[0m: \u001b[38;2;0;135;0m\"1.0\"\u001b[0m,                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"max_model_len\"\u001b[0m: \u001b[38;2;0;135;0m\"12000\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"aggregation\"\u001b[0m: \u001b[38;2;0;135;0m\"\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"postprocessing\"\u001b[0m: \u001b[38;2;0;135;0m\"False\"\u001b[0m                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"OutputDataConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"S3OutputPath\"\u001b[0m:                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"s3://mufi-test-serverless-smtj/eval/\"\u001b[0m,                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"CompressionType\"\u001b[0m: \u001b[38;2;0;135;0m\"NONE\"\u001b[0m                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"InputDataConfig\"\u001b[0m: \u001b[1m[\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ChannelName\"\u001b[0m: \u001b[38;2;0;135;0m\"train\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"DataSource\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"S3DataSource\"\u001b[0m: \u001b[1m{\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"S3DataType\"\u001b[0m: \u001b[38;2;0;135;0m\"S3Prefix\"\u001b[0m,                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"S3Uri\"\u001b[0m:                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t19\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m5443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\"\u001b[0m            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"AssociateLineage\"\u001b[0m,                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Lineage\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"DependsOn\"\u001b[0m: \u001b[1m[\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"CreateEvaluationAction\"\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m]\u001b[0m,                                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Artifacts\"\u001b[0m: \u001b[1m[\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ArtifactName\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m{\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m}\u001b[0m,                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"base-eval-report\"\u001b[0m                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m]\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ArtifactType\"\u001b[0m: \u001b[38;2;0;135;0m\"EvaluationReport\"\u001b[0m,                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m:                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"Steps.EvaluateBaseModel.OutputDataConfig.S3OutputPath\"\u001b[0m          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m,                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ArtifactName\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m{\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m}\u001b[0m,                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"custom-eval-report\"\u001b[0m                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m]\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ArtifactType\"\u001b[0m: \u001b[38;2;0;135;0m\"EvaluationReport\"\u001b[0m,                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m:                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"Steps.EvaluateCustomModel.OutputDataConfig.S3OutputPath\"\u001b[0m        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Associations\"\u001b[0m: \u001b[1m[\u001b[0m                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m,                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"base-eval-report\"\u001b[0m                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Artifact\"\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m: \u001b[1m{\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"/\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:ac\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mtion\"\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m,                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m,                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"custom-eval-report\"\u001b[0m                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Artifact\"\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m: \u001b[1m{\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"/\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:ac\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mtion\"\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[1m]\u001b[0m                                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[1m}\u001b[0m                                                                \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found existing pipeline:                                              <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#199\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">199</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-BenchmarkEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">c344c91d-6f62-4907-85cc-7e6b2</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">9171c42</span>                                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found existing pipeline:                                              \u001b]8;id=588942;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=925025;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#199\u001b\\\u001b[2m199\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-BenchmarkEvaluation-\u001b[93mc344c91d-6f62-4907-85cc-7e6b2\u001b[0m \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m9171c42\u001b[0m                                                               \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Updating pipeline                                                     <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#202\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">202</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-BenchmarkEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">c344c91d-6f62-4907-85cc-7e6b2</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">9171c42</span> with latest definition                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Updating pipeline                                                     \u001b]8;id=746487;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=234699;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#202\u001b\\\u001b[2m202\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-BenchmarkEvaluation-\u001b[93mc344c91d-6f62-4907-85cc-7e6b2\u001b[0m \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m9171c42\u001b[0m with latest definition                                        \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Updating pipeline resource.                                         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">resources.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py#30306\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">30306</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Updating pipeline resource.                                         \u001b]8;id=908194;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py\u001b\\\u001b[2mresources.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=233215;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py#30306\u001b\\\u001b[2m30306\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:40:22] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Successfully updated pipeline:                                        <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#208\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">208</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-BenchmarkEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">c344c91d-6f62-4907-85cc-7e6b2</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">9171c42</span>                                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:40:22]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Successfully updated pipeline:                                        \u001b]8;id=321336;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=381496;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#208\u001b\\\u001b[2m208\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-BenchmarkEvaluation-\u001b[93mc344c91d-6f62-4907-85cc-7e6b2\u001b[0m \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m9171c42\u001b[0m                                                               \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Starting pipeline execution: gen-qa-eval-demo-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1764452422</span>              <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#263\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">263</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Starting pipeline execution: gen-qa-eval-demo-\u001b[1;36m1764452422\u001b[0m              \u001b]8;id=359442;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=958972;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#263\u001b\\\u001b[2m263\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Pipeline execution started:                                           <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#274\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">274</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         -BenchmarkEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">c344c91d-6f62-4907-85cc-7e6b29171c42</span>/execution/9 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         5qr3e96dblb                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Pipeline execution started:                                           \u001b]8;id=73999;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=223527;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#274\u001b\\\u001b[2m274\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         -BenchmarkEvaluation-\u001b[93mc344c91d-6f62-4907-85cc-7e6b29171c42\u001b[0m/execution/9 \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         5qr3e96dblb                                                           \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">BenchmarkEvaluationExecution</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">arn</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-BenchmarkEvaluation-c344c91d-6f62-4907-85cc-7e6b29171c42/execution/95qr3e96dblb'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'gen-qa-eval-demo'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">PipelineExecutionStatus</span><span style=\"font-weight: bold\">(</span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">overall_status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>, <span style=\"color: #d7af00; text-decoration-color: #d7af00\">step_details</span>=<span style=\"font-weight: bold\">[]</span>, <span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">last_modified_time</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">datetime</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">.datetime</span><span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2025</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">29</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">40</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">22</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">284000</span>, <span style=\"color: #d7af00; text-decoration-color: #d7af00\">tzinfo</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">tzlocal</span><span style=\"font-weight: bold\">())</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">eval_type</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">EvalType.BENCHMARK:</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008700; text-decoration-color: #008700\">'benchmark'</span><span style=\"font-weight: bold\">&gt;</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">s3_output_path</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">steps</span>=<span style=\"font-weight: bold\">[]</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mBenchmarkEvaluationExecution\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0marn\u001b[0m=\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-BenchmarkEvaluation-c344c91d-6f62-4907-85cc-7e6b29171c42/execution/95qr3e96dblb'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'gen-qa-eval-demo'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[1;38;2;225;0;225mPipelineExecutionStatus\u001b[0m\u001b[1m(\u001b[0m\u001b[38;2;215;175;0moverall_status\u001b[0m=\u001b[38;2;0;135;0m'Executing'\u001b[0m, \u001b[38;2;215;175;0mstep_details\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m]\u001b[0m, \u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mlast_modified_time\u001b[0m=\u001b[1;38;2;225;0;225mdatetime\u001b[0m\u001b[1;38;2;225;0;225m.datetime\u001b[0m\u001b[1m(\u001b[0m\u001b[1;36m2025\u001b[0m, \u001b[1;36m11\u001b[0m, \u001b[1;36m29\u001b[0m, \u001b[1;36m13\u001b[0m, \u001b[1;36m40\u001b[0m, \u001b[1;36m22\u001b[0m, \u001b[1;36m284000\u001b[0m, \u001b[38;2;215;175;0mtzinfo\u001b[0m=\u001b[1;38;2;225;0;225mtzlocal\u001b[0m\u001b[1m(\u001b[0m\u001b[1m)\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0meval_type\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;38;2;225;0;225mEvalType.BENCHMARK:\u001b[0m\u001b[39m \u001b[0m\u001b[38;2;0;135;0m'benchmark'\u001b[0m\u001b[1m>\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0ms3_output_path\u001b[0m=\u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0msteps\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Pipeline Execution ARN: arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-BenchmarkEvaluation-c344c91d-6f62-4907-85cc-7e6b29171c42/execution/95qr3e96dblb\n",
      "Initial Status: Executing\n"
     ]
    }
   ],
   "source": [
    "# Run evaluation with configured parameters\n",
    "execution = evaluator.evaluate()\n",
    "pprint(execution)\n",
    "\n",
    "print(f\"\\nPipeline Execution ARN: {execution.arn}\")\n",
    "print(f\"Initial Status: {execution.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alternative: Override Subtasks at Runtime\n",
    "\n",
    "For benchmarks with subtask support, you can override subtasks when calling evaluate():"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Override subtasks at evaluation time\n",
    "# execution = mmlu_evaluator.evaluate(subtask=\"abstract_algebra\")  # Single subtask\n",
    "# execution = mmlu_evaluator.evaluate(subtask=[\"abstract_algebra\", \"anatomy\"])  # Multiple subtasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Monitor Execution\n",
    "\n",
    "Check the job status and refresh as needed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">PipelineExecutionStatus</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">overall_status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">step_details</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'EvaluateCustomModel'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:26:38.084000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">&lt;sagemaker.core.utils.utils.Unassigned</span><span style=\"color: #008700; text-decoration-color: #008700\"> object at 0x120de0b60&gt;'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">)</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'EvaluateBaseModel'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:26:38.083000-08:00'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'&lt;sagemaker.core.utils.utils.Unassigned object at 0x120de0b60</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">&gt;</span><span style=\"color: #008700; text-decoration-color: #008700\">'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'CreateEvaluationAction'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Succeeded'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:26:38.083000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:26:42.759000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mPipelineExecutionStatus\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0moverall_status\u001b[0m=\u001b[38;2;0;135;0m'Executing'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mstep_details\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'EvaluateCustomModel'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[38;2;0;135;0m'Executing'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-29T13:26:38.084000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m=\u001b[38;2;0;135;0m'\u001b[0m\u001b[1;38;2;0;135;0m<\u001b[0m\u001b[1;38;2;0;135;0msagemaker.core.utils.utils.Unassigned\u001b[0m\u001b[38;2;0;135;0m object at 0x120de0b60>'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1;39m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'EvaluateBaseModel'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'Executing'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'2025-11-29T13:26:38.083000-08:00'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'<sagemaker.core.utils.utils.Unassigned object at 0x120de0b60\u001b[0m\u001b[1;38;2;0;135;0m>\u001b[0m\u001b[38;2;0;135;0m'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'CreateEvaluationAction'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[38;2;0;135;0m'Succeeded'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-29T13:26:38.083000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-29T13:26:42.759000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Step Details:\n",
      "  EvaluateCustomModel: Executing\n",
      "  EvaluateBaseModel: Executing\n",
      "  CreateEvaluationAction: Succeeded\n"
     ]
    }
   ],
   "source": [
    "# Refresh status\n",
    "execution.refresh()\n",
    "\n",
    "# Display job status with step details\n",
    "pprint(execution.status)\n",
    "\n",
    "# Display individual step statuses\n",
    "if execution.status.step_details:\n",
    "    print(\"\\nStep Details:\")\n",
    "    for step in execution.status.step_details:\n",
    "        print(f\"  {step.name}: {step.status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Wait for Completion\n",
    "\n",
    "Wait for the pipeline to complete. This provides rich progress updates in Jupyter notebooks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─────────────────────────────────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Pipeline Execution Status</span><span style=\"color: #000080; text-decoration-color: #000080\"> ───────────────────────────────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Overall Status       </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\">Succeeded</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span>                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Target Status        </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\">Succeeded</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span>                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Elapsed Time         </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0.5s      </span>                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Pipeline Steps</span>                                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Step Name                       Status           Duration     </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> AssociateLineage               </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 3.3s         </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> EvaluateCustomModel            </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 3714.0s      </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> EvaluateBaseModel              </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 5366.2s      </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> CreateEvaluationAction         </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 2.7s         </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34mPipeline Execution Status\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m \u001b[0m\u001b[1;36mOverall Status      \u001b[0m\u001b[1;36m \u001b[0m\u001b[37m \u001b[0m\u001b[1;37mSucceeded\u001b[0m\u001b[37m \u001b[0m                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m \u001b[0m\u001b[1;36mTarget Status       \u001b[0m\u001b[1;36m \u001b[0m\u001b[37m \u001b[0m\u001b[1;37mSucceeded\u001b[0m\u001b[37m \u001b[0m                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m \u001b[0m\u001b[1;36mElapsed Time        \u001b[0m\u001b[1;36m \u001b[0m\u001b[37m \u001b[0m\u001b[37m0.5s     \u001b[0m\u001b[37m \u001b[0m                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;35mPipeline Steps\u001b[0m                                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mStep Name                     \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35mStatus         \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35mDuration    \u001b[0m\u001b[1;35m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mAssociateLineage              \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m3.3s        \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mEvaluateCustomModel           \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m3714.0s     \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mEvaluateBaseModel             \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m5366.2s     \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mCreateEvaluationAction        \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m2.7s        \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 16:21:20] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Final Resource Status: Succeeded                                      <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#979\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">979</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 16:21:20]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Final Resource Status: Succeeded                                      \u001b]8;id=401306;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=749;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#979\u001b\\\u001b[2m979\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Final Status: Succeeded\n"
     ]
    }
   ],
   "source": [
    "# Wait for job completion with progress updates\n",
    "# This will show a rich progress display in Jupyter\n",
    "execution.wait(target_status=\"Succeeded\", poll=5, timeout=3600)\n",
    "\n",
    "print(f\"\\nFinal Status: {execution.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: View Results\n",
    "\n",
    "Display the evaluation results in a formatted table:\n",
    "\n",
    "Output Structure:\n",
    "\n",
    "Evaluation results are stored in S3:\n",
    "\n",
    "```\n",
    "s3://your-bucket/output/\n",
    "└── job_name/\n",
    "    └── output/\n",
    "        └── output.tar.gz\n",
    "```\n",
    "\n",
    "Extract output.tar.gz to reveal:\n",
    "\n",
    "```\n",
    "run_name/\n",
    "├── eval_results/\n",
    "│   ├── results_[timestamp].json\n",
    "│   ├── inference_output.jsonl (for gen_qa)\n",
    "│   └── details/\n",
    "│       └── model/\n",
    "│           └── <execution-date-time>/\n",
    "│               └── details_<task_name>_#_<datetime>.parquet\n",
    "└── tensorboard_results/\n",
    "    └── eval/\n",
    "        └── events.out.tfevents.[timestamp]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/'</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/'\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 16:21:25] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> S3 bucket: mufi-test-serverless-smtj, prefix: eval           <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#130\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">130</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 16:21:25]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m S3 bucket: mufi-test-serverless-smtj, prefix: eval           \u001b]8;id=671086;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=908024;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#130\u001b\\\u001b[2m130\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Extracted training job name:                                  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#63\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">63</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         pipelines-95qr3e96dblb-EvaluateCustomModel-F51y8F3Pg7 from    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                        </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         step: EvaluateCustomModel                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                        </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Extracted training job name:                                  \u001b]8;id=813615;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=57499;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#63\u001b\\\u001b[2m63\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         pipelines-95qr3e96dblb-EvaluateCustomModel-F51y8F3Pg7 from    \u001b[2m                        \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         step: EvaluateCustomModel                                     \u001b[2m                        \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 16:21:26] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Extracted training job name:                                  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#63\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">63</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         pipelines-95qr3e96dblb-EvaluateBaseModel-VA9YzcdIVI from      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                        </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         step: EvaluateBaseModel                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                        </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 16:21:26]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Extracted training job name:                                  \u001b]8;id=745707;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=953308;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#63\u001b\\\u001b[2m63\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         pipelines-95qr3e96dblb-EvaluateBaseModel-VA9YzcdIVI from      \u001b[2m                        \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         step: EvaluateBaseModel                                       \u001b[2m                        \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for results_*.json in                              <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#150\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">150</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval/pipelines-95qr3e96dblb-E</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">valuateCustomModel-F51y8F3Pg7/output/output/</span>                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for results_*.json in                              \u001b]8;id=805603;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=739949;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#150\u001b\\\u001b[2m150\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/eval/pipelines-95qr3e96dblb-E\u001b[0m \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225mvaluateCustomModel-F51y8F3Pg7/output/output/\u001b[0m                 \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found results file:                                          <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#168\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">168</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         eval/pipelines-95qr3e96dblb-EvaluateCustomModel-F51y8F3Pg7/o <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         utput/output/eval-meta_textgeneration_llama_3_2_1b_instruct- <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         -or8pa/eval_results/results_2025-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span>-29T22-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">41</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">53.186048</span>+<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">00-00</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">.j</span>son                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found results file:                                          \u001b]8;id=188825;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=667854;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#168\u001b\\\u001b[2m168\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         eval/pipelines-95qr3e96dblb-EvaluateCustomModel-F51y8F3Pg7/o \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         utput/output/eval-meta_textgeneration_llama_3_2_1b_instruct- \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         -or8pa/eval_results/results_2025-\u001b[1;36m11\u001b[0m-29T22-\u001b[1;36m41\u001b[0m-\u001b[1;36m53.186048\u001b[0m+\u001b[1;36m00-00\u001b[0m \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[1;36m.j\u001b[0mson                                                        \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for results_*.json in                              <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#150\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">150</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval/pipelines-95qr3e96dblb-E</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">valuateBaseModel-VA9YzcdIVI/output/output/</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for results_*.json in                              \u001b]8;id=270113;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=844454;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#150\u001b\\\u001b[2m150\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/eval/pipelines-95qr3e96dblb-E\u001b[0m \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225mvaluateBaseModel-VA9YzcdIVI/output/output/\u001b[0m                   \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found results file:                                          <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#168\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">168</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         eval/pipelines-95qr3e96dblb-EvaluateBaseModel-VA9YzcdIVI/out <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         put/output/eval-meta_textgeneration_llama_3_2_1b_instruct--o <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         r8pa/eval_results/results_2025-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span>-29T23-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">09</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21.277725</span>+<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">00-00.j</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         son                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found results file:                                          \u001b]8;id=221667;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=736866;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#168\u001b\\\u001b[2m168\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         eval/pipelines-95qr3e96dblb-EvaluateBaseModel-VA9YzcdIVI/out \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         put/output/eval-meta_textgeneration_llama_3_2_1b_instruct--o \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         r8pa/eval_results/results_2025-\u001b[1;36m11\u001b[0m-29T23-\u001b[1;36m09\u001b[0m-\u001b[1;36m21.277725\u001b[0m+\u001b[1;36m00-00.j\u001b[0m \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         son                                                          \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using metrics from <span style=\"color: #008700; text-decoration-color: #008700\">'all'</span> key <span style=\"font-weight: bold\">(</span>standard benchmark format<span style=\"font-weight: bold\">)</span>      <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#93\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">93</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using metrics from \u001b[38;2;0;135;0m'all'\u001b[0m key \u001b[1m(\u001b[0mstandard benchmark format\u001b[1m)\u001b[0m      \u001b]8;id=431825;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=75452;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#93\u001b\\\u001b[2m93\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using metrics from <span style=\"color: #008700; text-decoration-color: #008700\">'all'</span> key <span style=\"font-weight: bold\">(</span>standard benchmark format<span style=\"font-weight: bold\">)</span>      <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#93\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">93</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using metrics from \u001b[38;2;0;135;0m'all'\u001b[0m key \u001b[1m(\u001b[0mstandard benchmark format\u001b[1m)\u001b[0m      \u001b]8;id=866976;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=697222;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#93\u001b\\\u001b[2m93\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                </span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold; font-style: italic\">Custom Model Results</span><span style=\"font-style: italic\">                </span>\n",
       "╭────────────────────────────────┬─────────────────╮\n",
       "│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> Metric                         </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">           Value </span>│\n",
       "├────────────────────────────────┼─────────────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> bleu                           </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          6.6928 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> bleu_stderr                    </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.7801 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> em                             </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">           1.23% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> em_stderr                      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0018 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1                             </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          19.04% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1_score_quasi                 </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          25.25% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1_score_quasi_stderr          </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0049 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1_stderr                      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> qem                            </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">           2.16% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> qem_stderr                     </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0024 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge1                         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          25.69% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge1_stderr                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge2                         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          19.09% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge2_stderr                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rougeL                         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          25.02% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rougeL_stderr                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "╰────────────────────────────────┴─────────────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                \u001b[0m\u001b[1;3;32mCustom Model Results\u001b[0m\u001b[3m                \u001b[0m\n",
       "╭────────────────────────────────┬─────────────────╮\n",
       "│\u001b[1;32m \u001b[0m\u001b[1;32mMetric                        \u001b[0m\u001b[1;32m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m          Value\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "├────────────────────────────────┼─────────────────┤\n",
       "│\u001b[36m \u001b[0m\u001b[36mbleu                          \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         6.6928\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mbleu_stderr                   \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.7801\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mem                            \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m          1.23%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mem_stderr                     \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0018\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1                            \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         19.04%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1_score_quasi                \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         25.25%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1_score_quasi_stderr         \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0049\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1_stderr                     \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mqem                           \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m          2.16%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mqem_stderr                    \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0024\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge1                        \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         25.69%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge1_stderr                 \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge2                        \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         19.09%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge2_stderr                 \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrougeL                        \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         25.02%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrougeL_stderr                 \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "╰────────────────────────────────┴─────────────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                 </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold; font-style: italic\">Base Model Results</span><span style=\"font-style: italic\">                 </span>\n",
       "╭────────────────────────────────┬─────────────────╮\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\"> Metric                         </span>│<span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">           Value </span>│\n",
       "├────────────────────────────────┼─────────────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> bleu                           </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          6.6928 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> bleu_stderr                    </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.7803 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> em                             </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">           1.29% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> em_stderr                      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0019 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1                             </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          19.09% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1_score_quasi                 </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          25.22% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1_score_quasi_stderr          </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0049 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1_stderr                      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> qem                            </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">           2.18% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> qem_stderr                     </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0024 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge1                         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          25.61% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge1_stderr                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge2                         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          19.04% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge2_stderr                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rougeL                         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          24.95% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rougeL_stderr                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "╰────────────────────────────────┴─────────────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                 \u001b[0m\u001b[1;3;33mBase Model Results\u001b[0m\u001b[3m                 \u001b[0m\n",
       "╭────────────────────────────────┬─────────────────╮\n",
       "│\u001b[1;33m \u001b[0m\u001b[1;33mMetric                        \u001b[0m\u001b[1;33m \u001b[0m│\u001b[1;33m \u001b[0m\u001b[1;33m          Value\u001b[0m\u001b[1;33m \u001b[0m│\n",
       "├────────────────────────────────┼─────────────────┤\n",
       "│\u001b[36m \u001b[0m\u001b[36mbleu                          \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         6.6928\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mbleu_stderr                   \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.7803\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mem                            \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m          1.29%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mem_stderr                     \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0019\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1                            \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         19.09%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1_score_quasi                \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         25.22%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1_score_quasi_stderr         \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0049\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1_stderr                     \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mqem                           \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m          2.18%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mqem_stderr                    \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0024\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge1                        \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         25.61%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge1_stderr                 \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge2                        \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         19.04%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge2_stderr                 \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrougeL                        \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         24.95%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrougeL_stderr                 \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "╰────────────────────────────────┴─────────────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─────────────────────────────────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Result Artifacts Location</span><span style=\"color: #000080; text-decoration-color: #000080\"> ───────────────────────────────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">📦 </span><span style=\"font-weight: bold\">Full evaluation artifacts available at:</span>                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Custom Model:</span>                                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #008080; text-decoration-color: #008080\">  s3://mufi-test-serverless-smtj/eval/pipelines-95qr3e96dblb-EvaluateCustomModel-F51y8F3Pg7/output/output/Non</span>  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #008080; text-decoration-color: #008080\">e/eval_results/</span>                                                                                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Base Model:</span>                                                                                                    <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #008080; text-decoration-color: #008080\">  s3://mufi-test-serverless-smtj/eval/pipelines-95qr3e96dblb-EvaluateBaseModel-VA9YzcdIVI/output/output/None/</span>  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #008080; text-decoration-color: #008080\">eval_results/</span>                                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34mResult Artifacts Location\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[1;34m📦 \u001b[0m\u001b[1mFull evaluation artifacts available at:\u001b[0m                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[1;32mCustom Model:\u001b[0m                                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[36m  s3://mufi-test-serverless-smtj/eval/pipelines-95qr3e96dblb-EvaluateCustomModel-F51y8F3Pg7/output/output/Non\u001b[0m  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[36me/eval_results/\u001b[0m                                                                                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[1;33mBase Model:\u001b[0m                                                                                                    \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[36m  s3://mufi-test-serverless-smtj/eval/pipelines-95qr3e96dblb-EvaluateBaseModel-VA9YzcdIVI/output/output/None/\u001b[0m  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[36meval_results/\u001b[0m                                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pprint(execution.s3_output_path)\n",
    "# Display results in a formatted table\n",
    "execution.show_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Retrieve an Existing Job\n",
    "\n",
    "You can retrieve and inspect any existing evaluation job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:35:47] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Extracted s3_output_path from training job                            <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#367\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">367</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         pipelines-inlsexrd7jes-EvaluateCustomModel-NuPrIoRW4Q:                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval/</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:35:47]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Extracted s3_output_path from training job                            \u001b]8;id=148252;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=588100;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#367\u001b\\\u001b[2m367\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         pipelines-inlsexrd7jes-EvaluateCustomModel-NuPrIoRW4Q:                \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/eval/\u001b[0m                                  \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">BenchmarkEvaluationExecution</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">arn</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-BenchmarkEvaluation-c344c91d-6f62-4907-85cc-7e6b29171c42/execution/inlsexrd7jes'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'inlsexrd7jes'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">PipelineExecutionStatus</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">overall_status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">step_details</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'EvaluateCustomModel'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:26:38.084000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">&lt;sagemaker.core.utils.utils.Unassigned</span><span style=\"color: #008700; text-decoration-color: #008700\"> object at 0x120de0b60&gt;'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">)</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'EvaluateBaseModel'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:26:38.083000-08:00'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'&lt;sagemaker.core.utils.utils.Unassigned object at 0x120de0b60&gt;'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">)</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'CreateEvaluationAction'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'Succeeded'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:26:38.083000-08:00'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:26:42.759000-08:00'</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">]</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">)</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">last_modified_time</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">datetime</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">.datetime</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2025</span><span style=\"color: #000000; text-decoration-color: #000000\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span><span style=\"color: #000000; text-decoration-color: #000000\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">29</span><span style=\"color: #000000; text-decoration-color: #000000\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13</span><span style=\"color: #000000; text-decoration-color: #000000\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">26</span><span style=\"color: #000000; text-decoration-color: #000000\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">37</span><span style=\"color: #000000; text-decoration-color: #000000\">, </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">300000</span><span style=\"color: #000000; text-decoration-color: #000000\">, </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">tzinfo</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">tzlocal</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">())</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">eval_type</span><span style=\"color: #000000; text-decoration-color: #000000\">=&lt;EvalType.BENCHMARK: </span><span style=\"color: #008700; text-decoration-color: #008700\">'benchmark'</span><span style=\"font-weight: bold\">&gt;</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">s3_output_path</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">steps</span>=<span style=\"font-weight: bold\">[]</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mBenchmarkEvaluationExecution\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0marn\u001b[0m=\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-BenchmarkEvaluation-c344c91d-6f62-4907-85cc-7e6b29171c42/execution/inlsexrd7jes'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'inlsexrd7jes'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[1;38;2;225;0;225mPipelineExecutionStatus\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[38;2;215;175;0moverall_status\u001b[0m=\u001b[38;2;0;135;0m'Executing'\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[38;2;215;175;0mstep_details\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'EvaluateCustomModel'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[38;2;0;135;0m'Executing'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-29T13:26:38.084000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m=\u001b[38;2;0;135;0m'\u001b[0m\u001b[1;38;2;0;135;0m<\u001b[0m\u001b[1;38;2;0;135;0msagemaker.core.utils.utils.Unassigned\u001b[0m\u001b[38;2;0;135;0m object at 0x120de0b60>'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1;39m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'EvaluateBaseModel'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'Executing'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'2025-11-29T13:26:38.083000-08:00'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'<sagemaker.core.utils.utils.Unassigned object at 0x120de0b60>'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1;39m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'CreateEvaluationAction'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'Succeeded'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'2025-11-29T13:26:38.083000-08:00'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m\u001b[39m=\u001b[0m\u001b[38;2;0;135;0m'2025-11-29T13:26:42.759000-08:00'\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1;39m)\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;39m]\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m\u001b[39m=\u001b[0m\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mlast_modified_time\u001b[0m\u001b[39m=\u001b[0m\u001b[1;38;2;225;0;225mdatetime\u001b[0m\u001b[1;38;2;225;0;225m.datetime\u001b[0m\u001b[1;39m(\u001b[0m\u001b[1;36m2025\u001b[0m\u001b[39m, \u001b[0m\u001b[1;36m11\u001b[0m\u001b[39m, \u001b[0m\u001b[1;36m29\u001b[0m\u001b[39m, \u001b[0m\u001b[1;36m13\u001b[0m\u001b[39m, \u001b[0m\u001b[1;36m26\u001b[0m\u001b[39m, \u001b[0m\u001b[1;36m37\u001b[0m\u001b[39m, \u001b[0m\u001b[1;36m300000\u001b[0m\u001b[39m, \u001b[0m\u001b[38;2;215;175;0mtzinfo\u001b[0m\u001b[39m=\u001b[0m\u001b[1;38;2;225;0;225mtzlocal\u001b[0m\u001b[1;39m(\u001b[0m\u001b[1;39m)\u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0meval_type\u001b[0m\u001b[39m=<EvalType.BENCHMARK: \u001b[0m\u001b[38;2;0;135;0m'benchmark'\u001b[0m\u001b[1m>\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0ms3_output_path\u001b[0m=\u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0msteps\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Status: Executing\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #ff0000; text-decoration-color: #ff0000\">╭─────────────────────────────── </span><span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold\">Traceback </span><span style=\"color: #ff7f7f; text-decoration-color: #ff7f7f; font-weight: bold\">(most recent call last)</span><span style=\"color: #ff0000; text-decoration-color: #ff0000\"> ────────────────────────────────╮</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">&lt;module&gt;</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">22</span>                                                                                   <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>                                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">19 </span>pprint(existing_execution)                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">20 </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">print</span>(<span style=\"color: #808000; text-decoration-color: #808000\">f\"\\nStatus: {</span>existing_execution.status.overall_status<span style=\"color: #808000; text-decoration-color: #808000\">}\"</span>)                              <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">21 </span>                                                                                            <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>22 <span style=\"font-weight: bold; text-decoration: underline\">existing_execution.show_results()</span>                                                           <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">23 </span>                                                                                            <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>                                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">tele</span> <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">metry_logging.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">175</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">wrapper</span>                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>                                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">172 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">\"sagemaker_session is not provided or not valid.\"</span>,                     <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">173 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>func_name,                                                             <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">174 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>)                                                                          <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>175 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">return</span> <span style=\"font-weight: bold; text-decoration: underline\">func(*args, **kwargs)</span>                                               <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">176 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>                                                                                   <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">177 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">return</span> wrapper                                                                     <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">178 </span>                                                                                           <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>                                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">exe</span> <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">cution.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">1223</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">show_results</span>                                                                   <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>                                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1220 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.refresh()                                                                    <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1221 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1222 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.status.overall_status != <span style=\"color: #808000; text-decoration-color: #808000\">\"Succeeded\"</span>:                                     <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>1223 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff; font-weight: bold; text-decoration: underline\">raise</span><span style=\"font-weight: bold; text-decoration: underline\"> </span><span style=\"color: #00ffff; text-decoration-color: #00ffff; font-weight: bold; text-decoration: underline\">ValueError</span><span style=\"font-weight: bold; text-decoration: underline\">(</span>                                                             <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1224 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; font-weight: bold; text-decoration: underline\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold; text-decoration: underline\">f\"Cannot show results. Execution status is '{</span><span style=\"color: #00ffff; text-decoration-color: #00ffff; font-weight: bold; text-decoration: underline\">self</span><span style=\"font-weight: bold; text-decoration: underline\">.status.overall_status</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold; text-decoration: underline\">}</span>  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1225 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; font-weight: bold; text-decoration: underline\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold; text-decoration: underline\">f\"Results are only available after successful execution. \"</span>                <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1226 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; font-weight: bold; text-decoration: underline\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold; text-decoration: underline\">f\"Use execution.wait() to wait for completion or check execution.status </span>  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">╰──────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold\">ValueError: </span>Cannot show results. Execution status is <span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>. Results are only available after successful \n",
       "execution. Use <span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">execution.wait</span><span style=\"font-weight: bold\">()</span> to wait for completion or check execution.status for details.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[38;2;255;0;0m╭─\u001b[0m\u001b[38;2;255;0;0m──────────────────────────────\u001b[0m\u001b[38;2;255;0;0m \u001b[0m\u001b[1;38;2;255;0;0mTraceback \u001b[0m\u001b[1;2;38;2;255;0;0m(most recent call last)\u001b[0m\u001b[38;2;255;0;0m \u001b[0m\u001b[38;2;255;0;0m───────────────────────────────\u001b[0m\u001b[38;2;255;0;0m─╮\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m in \u001b[92m<module>\u001b[0m:\u001b[94m22\u001b[0m                                                                                   \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m                                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m19 \u001b[0mpprint(existing_execution)                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m20 \u001b[0m\u001b[96mprint\u001b[0m(\u001b[33mf\u001b[0m\u001b[33m\"\u001b[0m\u001b[33m\\n\u001b[0m\u001b[33mStatus: \u001b[0m\u001b[33m{\u001b[0mexisting_execution.status.overall_status\u001b[33m}\u001b[0m\u001b[33m\"\u001b[0m)                              \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m21 \u001b[0m                                                                                            \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[31m❱ \u001b[0m22 \u001b[1;4mexisting_execution.show_results()\u001b[0m                                                           \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m23 \u001b[0m                                                                                            \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m                                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[2;33m/Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/\u001b[0m\u001b[1;33mtele\u001b[0m \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[1;33mmetry_logging.py\u001b[0m:\u001b[94m175\u001b[0m in \u001b[92mwrapper\u001b[0m                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m                                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m172 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0m\u001b[33m\"\u001b[0m\u001b[33msagemaker_session is not provided or not valid.\u001b[0m\u001b[33m\"\u001b[0m,                     \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m173 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0mfunc_name,                                                             \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m174 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m)                                                                          \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[31m❱ \u001b[0m175 \u001b[2m│   │   │   │   \u001b[0m\u001b[94mreturn\u001b[0m \u001b[1;4mfunc(*args, **kwargs)\u001b[0m                                               \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m176 \u001b[0m\u001b[2m│   │   \u001b[0m                                                                                   \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m177 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mreturn\u001b[0m wrapper                                                                     \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m178 \u001b[0m                                                                                           \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m                                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[2;33m/Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/\u001b[0m\u001b[1;33mexe\u001b[0m \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[1;33mcution.py\u001b[0m:\u001b[94m1223\u001b[0m in \u001b[92mshow_results\u001b[0m                                                                   \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m                                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1220 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[96mself\u001b[0m.refresh()                                                                    \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1221 \u001b[0m\u001b[2m│   │   \u001b[0m                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1222 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[96mself\u001b[0m.status.overall_status != \u001b[33m\"\u001b[0m\u001b[33mSucceeded\u001b[0m\u001b[33m\"\u001b[0m:                                     \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[31m❱ \u001b[0m1223 \u001b[2m│   │   │   \u001b[0m\u001b[1;4;94mraise\u001b[0m\u001b[1;4m \u001b[0m\u001b[1;4;96mValueError\u001b[0m\u001b[1;4m(\u001b[0m                                                             \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1224 \u001b[0m\u001b[1;2;4m│   │   │   │   \u001b[0m\u001b[1;4;33mf\u001b[0m\u001b[1;4;33m\"\u001b[0m\u001b[1;4;33mCannot show results. Execution status is \u001b[0m\u001b[1;4;33m'\u001b[0m\u001b[1;4;33m{\u001b[0m\u001b[1;4;96mself\u001b[0m\u001b[1;4m.status.overall_status\u001b[0m\u001b[1;4;33m}\u001b[0m  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1225 \u001b[0m\u001b[1;2;4m│   │   │   │   \u001b[0m\u001b[1;4;33mf\u001b[0m\u001b[1;4;33m\"\u001b[0m\u001b[1;4;33mResults are only available after successful execution. \u001b[0m\u001b[1;4;33m\"\u001b[0m                \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1226 \u001b[0m\u001b[1;2;4m│   │   │   │   \u001b[0m\u001b[1;4;33mf\u001b[0m\u001b[1;4;33m\"\u001b[0m\u001b[1;4;33mUse execution.wait() to wait for completion or check execution.status \u001b[0m  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
       "\u001b[1;91mValueError: \u001b[0mCannot show results. Execution status is \u001b[38;2;0;135;0m'Executing'\u001b[0m. Results are only available after successful \n",
       "execution. Use \u001b[1;38;2;225;0;225mexecution.wait\u001b[0m\u001b[1m(\u001b[0m\u001b[1m)\u001b[0m to wait for completion or check execution.status for details.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sagemaker.train.evaluate import EvaluationPipelineExecution\n",
    "from rich.pretty import pprint\n",
    "\n",
    "\n",
    "# Get an existing job by ARN\n",
    "# Replace with your actual pipeline execution ARN\n",
    "existing_arn = \"arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-BenchmarkEvaluation-c344c91d-6f62-4907-85cc-7e6b29171c42/execution/inlsexrd7jes\"\n",
    "\n",
    "# base model only example\n",
    "# existing_arn = \"arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-benchmark/execution/gdp9f4dbv2vi\"\n",
    "existing_execution = EvaluationPipelineExecution.get(\n",
    "    arn=existing_arn,\n",
    "    region=\"us-west-2\"\n",
    ")\n",
    "\n",
    "pprint(existing_execution)\n",
    "print(f\"\\nStatus: {existing_execution.status.overall_status}\")\n",
    "\n",
    "existing_execution.show_results()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO - sagemaker.modules.evaluate.benchmark_evaluator - Getting or creating artifact for source: arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28\n",
      "INFO - sagemaker.modules.evaluate.base_evaluator - Searching for existing artifact for model package: arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28\n",
      "INFO - sagemaker.modules.evaluate.base_evaluator - Found existing artifact: arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b3138877d772ec489bef\n",
      "INFO - sagemaker.modules.evaluate.benchmark_evaluator - Resolved model info - base_model_name: meta-textgeneration-llama-3-2-1b-instruct, base_model_arn: arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/Model/meta-textgeneration-llama-3-2-1b-instruct/1.10.0, source_model_package_arn: arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28\n",
      "INFO - sagemaker.modules.evaluate.benchmark_evaluator - Using configured hyperparameters: {'max_new_tokens': '8192', 'temperature': '0', 'top_k': '-1', 'top_p': '1.0', 'aggregation': '', 'postprocessing': 'False', 'max_model_len': '12000'}\n",
      "INFO - sagemaker.modules.evaluate.benchmark_evaluator - Using DETERMINISTIC_TEMPLATE for ModelPackage\n",
      "INFO - sagemaker.modules.evaluate.benchmark_evaluator - Resolved template parameters: {'role_arn': 'arn:aws:iam::052150106756:role/Admin', 'mlflow_resource_arn': 'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment', 'mlflow_experiment_name': None, 'mlflow_run_name': None, 'model_package_group_arn': 'arn:aws:sagemaker:us-west-2:052150106756:model-package-group/example-name-aovqo', 'source_model_package_arn': 'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28', 'base_model_arn': 'arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/Model/meta-textgeneration-llama-3-2-1b-instruct/1.10.0', 'task': 'gen_qa', 'strategy': 'gen_qa', 'evaluation_metric': 'all', 's3_output_path': 's3://mufi-test-serverless-smtj/eval/', 'dataset_uri': 's3://sagemaker-us-west-2-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl', 'subtask': '', 'pipeline_name': 'SagemakerEvaluation-Deterministic', 'dataset_artifact_arn': 'arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b3138877d772ec489bef', 'action_arn_prefix': 'arn:aws:sagemaker:us-west-2:052150106756:action', 'evaluate_base_model': True, 'max_new_tokens': '8192', 'temperature': '0', 'top_k': '-1', 'top_p': '1.0', 'aggregation': '', 'postprocessing': 'False', 'max_model_len': '12000'}\n",
      "INFO - sagemaker.modules.evaluate.execution - Found existing pipeline: SagemakerEvaluation-benchmark\n",
      "INFO - sagemaker.modules.evaluate.execution - Updating pipeline SagemakerEvaluation-benchmark with latest definition for benchmark\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/22/25 12:24:36] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Updating pipeline resource.                                         <a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/resources.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">resources.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/resources.py#30485\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">30485</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/22/25 12:24:36]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Updating pipeline resource.                                         \u001b]8;id=707103;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/resources.py\u001b\\\u001b[2mresources.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=260368;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/resources.py#30485\u001b\\\u001b[2m30485\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO - sagemaker_core.main.resources - Updating pipeline resource.\n",
      "INFO - sagemaker.modules.evaluate.execution - Successfully updated pipeline: SagemakerEvaluation-benchmark\n",
      "INFO - sagemaker.modules.evaluate.execution - Starting pipeline execution: gen-qa-eval-demo-1763843077\n",
      "INFO - sagemaker.modules.evaluate.execution - Pipeline execution started: arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-benchmark/execution/gv93gtwgr7w8\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">BenchmarkEvaluationExecution</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">arn</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-benchmark/execution/gv93gtwgr7w8'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'gen-qa-eval-demo'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">PipelineExecutionStatus</span><span style=\"font-weight: bold\">(</span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">overall_status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>, <span style=\"color: #d7af00; text-decoration-color: #d7af00\">step_details</span>=<span style=\"font-weight: bold\">[]</span>, <span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">last_modified_time</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">datetime</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">.datetime</span><span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2025</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">22</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">12</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">24</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">37</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">828000</span>, <span style=\"color: #d7af00; text-decoration-color: #d7af00\">tzinfo</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">tzlocal</span><span style=\"font-weight: bold\">())</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">eval_type</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">EvalType.BENCHMARK:</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008700; text-decoration-color: #008700\">'benchmark'</span><span style=\"font-weight: bold\">&gt;</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">s3_output_path</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">steps</span>=<span style=\"font-weight: bold\">[]</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mBenchmarkEvaluationExecution\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0marn\u001b[0m=\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-benchmark/execution/gv93gtwgr7w8'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'gen-qa-eval-demo'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[1;38;2;225;0;225mPipelineExecutionStatus\u001b[0m\u001b[1m(\u001b[0m\u001b[38;2;215;175;0moverall_status\u001b[0m=\u001b[38;2;0;135;0m'Executing'\u001b[0m, \u001b[38;2;215;175;0mstep_details\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m]\u001b[0m, \u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mlast_modified_time\u001b[0m=\u001b[1;38;2;225;0;225mdatetime\u001b[0m\u001b[1;38;2;225;0;225m.datetime\u001b[0m\u001b[1m(\u001b[0m\u001b[1;36m2025\u001b[0m, \u001b[1;36m11\u001b[0m, \u001b[1;36m22\u001b[0m, \u001b[1;36m12\u001b[0m, \u001b[1;36m24\u001b[0m, \u001b[1;36m37\u001b[0m, \u001b[1;36m828000\u001b[0m, \u001b[38;2;215;175;0mtzinfo\u001b[0m=\u001b[1;38;2;225;0;225mtzlocal\u001b[0m\u001b[1m(\u001b[0m\u001b[1m)\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0meval_type\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;38;2;225;0;225mEvalType.BENCHMARK:\u001b[0m\u001b[39m \u001b[0m\u001b[38;2;0;135;0m'benchmark'\u001b[0m\u001b[1m>\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0ms3_output_path\u001b[0m=\u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0msteps\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Pipeline Execution ARN: arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-benchmark/execution/gv93gtwgr7w8\n",
      "Initial Status: Executing\n"
     ]
    }
   ],
   "source": [
    "# Run evaluation with configured parameters\n",
    "execution = evaluator.evaluate()\n",
    "pprint(execution)\n",
    "\n",
    "print(f\"\\nPipeline Execution ARN: {execution.arn}\")\n",
    "print(f\"Initial Status: {execution.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: List All Benchmark Evaluations\n",
    "\n",
    "Retrieve all benchmark evaluation executions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:41:19] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Extracted s3_output_path from training job                            <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#367\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">367</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         pipelines-95qr3e96dblb-EvaluateCustomModel-F51y8F3Pg7:                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval/</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:41:19]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Extracted s3_output_path from training job                            \u001b]8;id=166943;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=816278;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#367\u001b\\\u001b[2m367\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         pipelines-95qr3e96dblb-EvaluateCustomModel-F51y8F3Pg7:                \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/eval/\u001b[0m                                  \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Extracted s3_output_path from training job                            <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#367\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">367</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         pipelines-inlsexrd7jes-EvaluateCustomModel-NuPrIoRW4Q:                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval/</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Extracted s3_output_path from training job                            \u001b]8;id=521868;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=351282;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#367\u001b\\\u001b[2m367\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         pipelines-inlsexrd7jes-EvaluateCustomModel-NuPrIoRW4Q:                \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/eval/\u001b[0m                                  \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2 evaluation(s)\n",
      "\n",
      "  95qr3e96dblb: Executing\n",
      "  inlsexrd7jes: Executing\n"
     ]
    }
   ],
   "source": [
    "# Get all benchmark evaluations (returns iterator)\n",
    "all_executions_iter = BenchMarkEvaluator.get_all(region=\"us-west-2\")\n",
    "all_executions = list(all_executions_iter)\n",
    "\n",
    "print(f\"Found {len(all_executions)} evaluation(s)\\n\")\n",
    "for exec in all_executions[:5]:  # Show first 5\n",
    "    print(f\"  {exec.name}: {exec.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9: Stop a Running Job (Optional)\n",
    "\n",
    "You can stop a running evaluation if needed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/shapes.py:2350: UserWarning: Field name \"schema\" in \"AutoMLSnowflakeDatasetDefinition\" shadows an attribute in parent \"Base\"\n",
      "  class AutoMLSnowflakeDatasetDefinition(Base):\n",
      "/Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/shapes.py:6372: UserWarning: Field name \"schema\" in \"SnowflakeDatasetDefinition\" shadows an attribute in parent \"Base\"\n",
      "  class SnowflakeDatasetDefinition(Base):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /Library/Application Support/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /Users/mufi/Library/Application Support/sagemaker/config.yaml\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/22/25 18:32:01] </span><span style=\"color: #d7af00; text-decoration-color: #d7af00; font-weight: bold\">WARNING </span> No boto3 session provided. Creating a new session.                        <a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/utils.py#339\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">339</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/22/25 18:32:01]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m No boto3 session provided. Creating a new session.                        \u001b]8;id=549422;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=573139;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/utils.py#339\u001b\\\u001b[2m339\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #d7af00; text-decoration-color: #d7af00; font-weight: bold\">WARNING </span> No config provided. Using default config.                                 <a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/utils.py#347\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">347</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m No config provided. Using default config.                                 \u001b]8;id=278829;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=978800;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/sagemaker_core/main/utils.py#347\u001b\\\u001b[2m347\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Succeeded\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AWS service error when stopping pipeline execution: Pipeline execution with ARN arn:aws:sagemaker:us-west-2:052150106756:pipeline/sagemakerevaluation-benchmark/execution/7rr30o7c2qfb status 'Succeeded'. Only pipelines with 'Executing' status can be stopped.\n"
     ]
    }
   ],
   "source": [
    "# Uncomment to stop the job\n",
    "# existing_execution.stop()\n",
    "# print(f\"Execution stopped. Status: {execution.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the Pipeline Structure\n",
    "\n",
    "The rendered pipeline definition includes:\n",
    "\n",
    "**4 Steps:**\n",
    "1. **CreateEvaluationAction** (Lineage): Sets up tracking\n",
    "2. **EvaluateBaseModel** (Training): Evaluates base model\n",
    "3. **EvaluateCustomModel** (Training): Evaluates custom model\n",
    "4. **AssociateLineage** (Lineage): Links results\n",
    "\n",
    "**Key Features:**\n",
    "- Template-based: Uses Jinja2 for flexible pipeline generation\n",
    "- Parallel execution: Base and custom models evaluated simultaneously\n",
    "- Serverless: No need to manage compute resources\n",
    "- MLflow integration: Automatic experiment tracking\n",
    "- Lineage tracking: Full traceability of evaluation artifacts\n",
    "\n",
    "**Typical Execution Time:**\n",
    "- Total: ~10-12 minutes\n",
    "- Downloading phase: ~5-7 minutes (model and dataset)\n",
    "- Training phase: ~3-5 minutes (running evaluation)\n",
    "- Lineage steps: ~2-4 seconds each"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
